Depth sensing technology can be used to determine a person's location in relation to nearby objects or to generate an image of a person's immediate environment in three dimensions (3D). One application in which depth sensing technology may be used is in head-mounted display (HMD) devices and other types of near-eye display (NED) devices. Depth sensing technology can employ a time-of-flight (ToF) depth camera. With ToF based depth sensing technology, a light source emits light onto nearby objects, and a ToF camera captures the light after it reflects off surfaces of the objects. The time taken for the light to travel from the light source and to reflect back from an object to the ToF camera is converted, based on the known speed of light, into a depth measurement (i.e., the distance to the object). Such a measurement can be processed with other similar measurements to create a map of physical surfaces in the user's environment (called a depth image or depth map) and, if desired, to render a 3D image of the user's environment.